storage: avoid excessive walking of the auxiliary directory #158160
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Member
|
This change is |
This comment was marked as resolved.
This comment was marked as resolved.
github-actions
bot
added
the
o-AI-Review-Potential-Issue-Detected
Pebble incrementally accounts its disk usage and this accounting is used to cheaply service Capacity calls. The auxiliary directory is a subdirectory of the data directory, and various Cockroach subsystems write to it directly without any incremental accounting of their disk usage. As a result, Capacity calculations walk the auxiliary directory to calculate the directory's disk usage anew each time. Previously, this calculation occurred during every call to Capacity. This commit adapts Pebble.Capacity to cache the computed capacity and recalculate it at most every minute. Ideally, we would incrementally account disk usage within the auxiliary directory (cockroachdb#96344). This change is a stopgap, avoiding the bulk of the CPU usage incurred by Capacity recalculations during IMPORTs. Epic: none Release note: none
RaduBerinde
reviewed
Nov 21, 2025
Member
RaduBerinde
left a comment
RaduBerinde
left a comment
There was a problem hiding this comment.
Reviewable status:
complete! 0 of 0 LGTMs obtained
pkg/storage/pebble.go line 812 at r1 (raw file):
auxiliarySize struct { mu syncutil.Mutex computedAt time.Time
[nit] crtime.Mono
pkg/storage/pebble.go line 1991 at r1 (raw file):
// Special-case: if the store-dir is configured using the root of some fs, // e.g. "/mnt/db", we might have special fs-created files like lost+found // that we can't read, so just ignore them rather than crashing.
[nit] s/crashing/erroring out
RaduBerinde
approved these changes
Nov 21, 2025
Member
RaduBerinde
left a comment
RaduBerinde
left a comment
There was a problem hiding this comment.
Reviewable status:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pebble incrementally accounts its disk usage and this accounting is used to cheaply service Capacity calls. The auxiliary directory is a subdirectory of the data directory, and various Cockroach subsystems write to it directly without any incremental accounting of their disk usage. As a result, Capacity calculations walk the auxiliary directory to calculate the directory's disk usage anew each time.
Previously, this calculation occurred during every call to Capacity. This commit adapts Pebble.Capacity to cache the computed capacity and recalculate it at most every minute. Ideally, we would incrementally account disk usage within the auxiliary directory (#96344). This change is a stopgap, avoiding the bulk of the CPU usage incurred by Capacity recalculations during IMPORTs.
Epic: none
Release note: none